Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval
流れ
1. a lot of 等のquantificational modifiersの修正
2. 代名詞の解釈
3. 複数名詞への対応 → ノードを増やす
②rule-based or classifier-basedなparserでsemantic graphからobject, relations, attributesを抽出
rule-based parserはsegmexを使用(後述)
classifier-based parser
objectのclassとrelationを予測
③MAP推定により, objectとbboxの対応関係を推定&スコア化し, image retrievalを実行
rule-based parser
9つのルールを定義
Adjectival modifiers
形容詞的修飾語
Subject-predicate-object constructions and subject-predicate constructions without an object
主語-述語-目的語構文と 主語と述語の構文で、目的語がない場合 オブジェクトのない主語述語構文
Copular constructions
Prepositional phrases
前置詞句
Possessive constructions
所有格の構文
Passive constructions
受動構文
Clausal modifiers of nouns
名詞の節付け修飾語
具体的には下のようにsegmexで定義されている
code:parser.java
/* A man is riding a horse. */
public static SemgrexPattern SUBJ_PRED_OBJ_TRIPLET_PATTERN = SemgrexPattern.compile("{}=pred >nsubj {tag:/NNP?S?/}=subj >/(iobj|dobj|nmod:.*)/=objreln {tag:/NNP?S?/}=obj !> cop {}");
/* A woman is smiling. */
public static SemgrexPattern SUBJ_PRED_PAIR_PATTERN = SemgrexPattern.compile("{}=pred >nsubj {tag:/NNP?S?/}=subj !>/(iobj|dobj|nmod:.*)/ {tag:/NNP?S?/} !>cop {}");
/* The man is a rider. */
public static SemgrexPattern COPULAR_PATTERN = SemgrexPattern.compile("{}=pred >nsubj {tag:/NNP?S?/}=subj >cop {}");
/* A smart woman. */
public static SemgrexPattern ADJ_MOD_PATTERN = SemgrexPattern.compile("{}=obj >/(amod)/ {}=adj");
/* The man is tall. */
public static SemgrexPattern ADJ_PRED_PATTERN = SemgrexPattern.compile("{tag:/J.*/}=adj >nsubj {}=obj");
/* A woman is in the house. */
public static SemgrexPattern PP_MOD_PATTERN = SemgrexPattern.compile("{tag:/NNP?S?/}=gov >/nmod:.*/=reln {}=mod");
/* His watch. */
public static SemgrexPattern POSS_PATTERN = SemgrexPattern.compile("{tag:/NNP?S?/}=gov >/nmod:poss/=reln {tag:/NNP?S?/}=mod");
/* */
public static SemgrexPattern AGENT_PATTERN = SemgrexPattern.compile("{tag:/V.*/}=pred >/nmod:agent/=reln {tag:/NNP?S?/}=subj >nsubjpass {tag:/NNP?S?/}=obj ");
/* A cat sitting in a chair. */
public static SemgrexPattern ACL_PATTERN = SemgrexPattern.compile("{}=subj >acl ({tag:/V.*/}=pred >/(iobj|dobj|nmod:.*)/=objreln {tag:/NNP?S?/}=obj)");